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Abstract 

We study codes on graphs combined with an iterative message passing algorithm 
for quantization. Specifically, we consider the binary erasure quantization (BEQ) 



^ . problem which is the dual of the binary erasure channel (BEC) coding problem. 

We show that duals of capacity achieving codes for the BEC yield codes which 
approach the minimum possible rate for the BEQ. In contrast, low density parity 
check codes cannot achieve the minimum rate unless their density grows at least 
logarithmically with block length. Furthermore, we show that duals of efficient 
^ ' iterative decoding algorithms for the BEC yield efficient encoding algorithms for 

O ■ the BEQ. Hence our results suggest that graphical models may yield near optimal 

codes in source coding as well as in channel coding and that duality plays a key 
role in such constructions. 
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Tjj- , Researchers have discovered that error correction codes defined on sparse graphs can be 

O I iteratively decoded with low complexity and vanishing error probability at rates close 

^ ■ to the Shannon limit. Based on the close parallels between error correction and data 

compression, we believe that similar graphical codes can approach the fundamental limits 
of data compression with reasonable complexity. Unfortunately, the existing suboptimal 
channel decoding algorithms for graphical codes generally fail unless the decoder input 
I is already near a codeword. Since this is usually not the case in source coding, either a 

new type of graph or a new suboptimal algorithm (or both) is required. 

Before developing iterative quantization techniques it is worth investigating the po- 
tential gains of such an approach over existing systems. For asymptotically high rates, 
when compressing a continuous source with finite moments relative to mean square error 
(MSE) distortion, entropy coded scalar quantization (ECSQ) is 1.53 dB from the rate- 
distortion limit . For moderate rates the gap is larger: in quantizing a Gaussian source 
relative to MSE distortion, ECSQ systems are 1.6-3.4 dB away from the rate-distortion 
limit. For these parameters, trellis coded quantization (TCQ) using a 256-state code with 
optimal quantization has a gap of 0.5-1.4 dB [2]. For higher rates, sources with larger 
tails {e.g., a source with a Laplacian distribution), or sources with memory, the gaps 
are larger. Thus for memoryless sources quantized at moderate rates, new codes have 
the potential to improve performance by the noticeable margin of a few decibels. More 
generally, the codes on graphs paradigm may prove valuable in a variety of scenarios 
involving speech, audio, video and other complicated sources. 



To illustrate possible approaches to developing graphical codes we focus on the binary 
erasure quantization (BEQ) problem which is the dual of the binary erasure channel 
(BEC) coding problem. First we describe the BEQ problem model in Section |21 Next, 
in Section El we present our main result for the BEQ: duals of low density parity check 
codes can be analyzed, encoded, and decoded by dualizing the corresponding techniques 
for the BEC. Specifically, by dualizing capacity achieving codes for the BEC we obtain 
rate-distortion approaching codes for the BEQ. Finally, we close with some concluding 
remarks in Section EJ 



2 Quantization Model 

Vectors and sequences are denoted with an arrow {e.g., x). Random variables or random 
vectors are denoted using the sans serif font {e.g., x or x). We consider the standard 
(memoryless) data compression problem and represent an instance of the problem with 
the tuple {S , Ps{s) , d{- , ■)) where S represents the source alphabet, Ps{s) represents the 
source distribution, and d{-, ■) represents a distortion measure. Specifically, a source s 
consists of a sequence of n random variables Si, S2, . . ., s„ each taking values in S and 
generated according to the distribution pg{s) = Y['i=iPs{si)- ^^'^^ R encoder /(■) maps 
s to an integer in {1, 2, ... , 2"^}, and the corresponding decoder g{-) maps the resulting 
integer into a reconstruction q. Distortion between the source s and the reconstruction 
q is measured via D = - Ym=i '^(^«' ^i)- 

Shannon derived the minimum possible rate required by any data compression system 
operating with distortion D. The so-called rate-distortion function is given by the formula 

R{D) = min /(s; q) (1) 

p,|,(g|s):i?[d(s,q)]<D 

where /(■; ■) denotes mutual information and E[] denotes expectation. 
2.1 Binary Erasure Quantization 

To highlight connections between error correction and data compression, we consider the 
binary erasure quantization (BEQ) problem where the source vector consists of ones, 
zeros, and "erasures" represented by the symbol *. Neither ones nor zeros may be 
changed, but erasures may be quantized to either zero or one. Practically, erasures may 
represent source samples which are missing, irrelevant, or corrupted by noise and so do 
not affect the distortion regardless of the value they are assigned. Formally, the BEQ 
problem with erasure probability e corresponds to 

5 = {0,1,*} (2a) 

Ps{s) = ^ ■ 5{s) + ^.S{s-l) + e-S{s-*) (2b) 

d{a, b) = if a = * or a = b, and 1 otherwise. (2c) 

It is straightforward to show that for £) = the distribution 

P^\s{q\s) = 6{q -s)ifse {0, 1}, and i ■ 6{q) + ^ ■ 6{q - 1) if s = *. (3) 

optimizes ([Q) and yields the value of the rate-distortion function at D = 0: 

RBEq{D = 0) = I - e. (4) 
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3 Codes For Erasure Quantization 



It is well-known that the encoder for a quantizer serves a similar function to the decoder 
for an error correcting code in the sense that both take a vector input {i.e., a source to 
quantize or channel output to decode) and map the result to bits {i.e., the compressed 
source or the transmitted message). The decoder for a quantizer can similarly be identi- 
fied with the encoder for an error correcting code in the sense that both take bits as input 
and produce a vector {i.e., a source reconstruction or a channel input). Thus it is natural 
to investigate whether swapping the encoder and decoder for a good error correcting code 
such as a low density parity check (LDPC) code produces a good quantizer. 

3.1 LDPC Codes Are Bad Quantizers 

One benefit of studying the BEQ problem is that it demonstrates why low density parity 
check (LDPC) codes are inherently unsuitable for quantization. Specifically, consider an 
LDPC code like the one illustrated in Fig. fusing Forney's normal graph notation 0. If 
all the variables connected to a given check are not erased, then there is an even chance 
that no code symbol can match the source in that position and thus the distortion will 
be positive regardless of the code rate. Thus, as stated in Theorem [T] and proved in 
Appendix El successful decoding is asymptotically unlikely unless the density of every 
parity check matrix for the code increases logarithmically with the block length.^ 

Theorem 1. Let C(n) be a sequence of linear codes of length n and fixed rate R such that 
the probability that binary erasure quantization using C(^n) of a random source sequence 
with e ■ n erasures will succeed with zero distortion is bounded away from as n ^ oo. 
Then regardless of the values of R and e, the degree of the parity-check nodes in any 
parity-check graph representation of C(n) niust increase at least logarithmically with n. 




Figure 1: Using an LDPC code for binary erasure quantization. The boxes with = signs 
are repetition nodes: all edges connected to an = box must have the same value. The 
boxes with + signs are check nodes: the modulo 2 sum of the values on edges connected 
to a + box must be 0. The source consists of O's, I's, and erasures represented by *'s. 
Erasures may be quantized to or 1 while incurring no distortion. A non-zero distortion 
must be incurred for the source shown above since the left-most check cannot be satisfied. 



The poor performance of LDPC codes for quantization may seem surprising in light 
of their excellent properties in channel coding, but it has long been recognized that good 

"'^We may expect the density of a code to increase as it approaches the capacity or rate- distortion 
function, but a code whose density also increases with block length seems undesirable. 
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codes for error correction and quantization may be different. The former is essentially 
a packing problem where the goal is to place as many codewords as possible in a given 
space such that the codewords are far apart and can be distinguished despite noise. The 
latter is a covering problem where the goal is to place as few codewords as possible in a 
given space such that every point in space is near at least one codeword. 

From any good error correcting code (respectively data compression code) it is easy 
to obtain another code which is almost as good at error correction (resp. data compres- 
sion) but terrible at source coding (resp. channel coding). For example, removing half 
the codewords has an asymptotically negligible effect on error correction since it only 
decreases the rate by 1/n and only increases robustness. But, removing half the code- 
words can dramatically hinder source coding since half the time the source may be very 
far from the nearest codeword. Conversely, doubling the number of codewords has an 
asymptotically negligible effect on data compression since the rate only increases by 
while the distortion may decrease slightly. But doubling the number of codewords can be 
catastrophic for error correction if it drastically reduces the distance between codewords. 

3.2 Dual LDPC Codes 

Many researchers have explored duality relationships between error correction and source 
coding. Such work demonstrates that often a good solution for one problem can be 
obtained by dualizing a good solution to the other. Continuing in this tradition, we 
study the properties of dual LDPC codes for binary erasure quantization. 

Formally, a length n binary linear code C is a subspace of the n dimensional vector 
space over the binary field and the dual code C"*- is the subspace orthogonal to C. For 
LDPC codes, the code C is usually specified by the parity check matrix H representing 
the constraint that Har^ = if and only if a; is a codeword. To obtain the dual code 
C"*" we can recall that the generator G"*" of C"*" is exactly H. If the code C is represented 
by a normal graph as in Fig. ^ then the graph of the dual code C can be obtained by 
swapping -|- and = nodes . In dualizing the code graph in this manner it may be useful 
to note that while the graph of C obtained from H represents a syndrome former for C, 
the dualized graph represents an encoder for C"*". 

For example. Fig. |21 is obtained by dualizing the code graph in Fig. [T] Notice that 
while the original code cannot quantize the source with distortion 0, the dual code can. 
Intuitively, the advantage of a low density encoder structure is that it provides a simple 
representation of a basis which can be used to construct the desired vector. In the 
following sections, we investigate the properties of dual LDPC codes for quantization 
with both optimal quantization and iterative quantization. 

3.3 Optimal Quantization/Decoding and Duality 

The following theorem (proved in Appendix ^ demonstrates the dual relationship be- 
tween channel decoding and source quantization using optimal decoding/quantization 
algorithms. 

Theorem 2. A channel decoder for the code C can correctly decode every received se- 
quence with the erasure pattern^ e if and only if a quantizer for the code can success- 
fully quantize every^ source sequence with the erasure pattern = \ — e. 

^If symbol i is erased (resp. unerased) then = 1 (resp. Cj = 0) in our notation for erasure patterns. 
•^Note that some source sequences (e.i?., the aU zero sequence) can be successfully quantized using 
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Figure 2: Using the dual of an LDPC code for binary erasure quantization. Choosing 
values for the variables at the bottom produces a codeword. The values for each sample 
of the resulting codeword are obtained by taking the sum modulo 2 of the connected 
variables. In contrast to Fig. ^ this structure can successfully match the source with no 
distortion if the bottom 3 variables are set to 0, 0, 1. 



From this result we immediately obtain the following Corollary. 

Corollary 1. Let C(^n) be a sequence of linear codes which achieves the capacity of a 
binary erasure channel with erasure probability e using optimal decoding. The sequence 
obtained by taking the duals of C[n) achieves the minimum rate for D = for the 
BEQ with erasure probability e-*" = 1 — e using optimal quantization. 

The statement and proof of the two preceding results contain a curious duality be- 
tween erased/known symbols in source coding and known/erased symbols in channel 
coding. A similar duality exists between a likelihood ratio, A, and its Fourier transform 
A = used in dualizing the sum-product algorithm |3J pp. 545-546]. Specifically, 
the Fourier transform maps known/erased likelihood ratios to erased/known likelihood 
ratios. 

3.4 Iterative Decoding/Quantization and Duality 

In the following we first review the intuition behind iterative erasure decoding algorithms 
and describe the particular decoding algorithm we consider in Table ^ Next we outline 
the intuition behind a similar approach for iterative quantization and precisely describe 
our quantization algorithm in Table El Finally, we show that these algorithms are duals. 

3.4.1 Iterative Erasure Decoding 

Many iterative message-passing decoding algorithms are essentially based on the following 
idea. The outgoing message on edge z of a + node is the modulo-2 sum of all incoming 
messages (excluding edge i) with the proviso that if any incoming message (excluding 
edge i) is * then the outgoing message is also *. For an = node, the outgoing message on 
edge 2 is * only if all other incoming messages are *, otherwise the outgoing message is 
the same as the known incoming message or messages. These message-passing rules can 

regardless of . Similarly, a system which decodes ambiguous received sequences to the all zero 
sequence may succeed even when many erasures occur. Thus to obtain the desired equivalence between 
correct decoding and successful quantization we define correct decoding (resp. successful quantization) 
as being able to deduce the transmitted codeword (resp. a codeword matching non-erased positions of 
the source) for every possible received sequence (resp. source) with the the erasure pattern e (resp. e^). 
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be interpreted as determining the outgoing message on edge i by applying the following 
"sum" and "product" formulas to all other incoming messages. ^ 
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(5) 



It is well-known that such algorithms yield optimal decoding on a tree and also 
perform well on graphs with cycles provided appropriate scheduling and initialization 
rules are selected. Initializing all messages to * and using sequential or parallel schedules 
are common choices. For the purpose of proving theorems, we consider a sequential 
schedule in the ERASURE-DECODE algorithm of Tabled 

Table 1: An algorithm for iteratively decoding data with erasures. 



ERASURE-DECODE(H,y) 

1: while y has at least one erased sample do 

2: if 3 row z of H (i.e., a check) connected to exactly one erased variable yj then 

3: Set yj to be the XOR of all unerased bits in the check 

4: else 

5: return FAIL 

6: end if 

7: end while 

8: Set X to the message variables obtained from y 

9: return the message variables x 



3.4.2 Iterative Erasure Quantization 



The message-passing rules in (0) can also be applied to the BEQ problem for graphs 
without cycles provided some form of tie-breaking is used. Specifically, some variables 
will receive erasure messages even after the algorithm has completed. Such variables 
can be arbitrarily chosen to be either or 1 and still produce a valid quantization. For 
example in quantizing the source (*,*,!) with a (3,2) single parity check code, both 
(1, 0, 1) and (0, 1, 1) are equally valid results and this tie can broken arbitrarily. 

On a graph with cycles, however, generalizing this approach by initializing all un- 
known messages to * usually fails. For example, on the dual of a Gallager code or a 
code like the one represented in Fig. |21 such an initialization rule leads to all messages 
being erased at every step of the algorithm. To perform effective tie-breaking, we need to 
distinguish between variables which can be arbitrarily set to or 1 and variables which 
have not yet received enough information to be determined. 

One way to distinguish between these cases is to denote the former as erasures with 
the symbol * and the latter as null messages with the symbol 0, and initialize all messages 
to 0. With this initialization, we can use the following "sum" and "product" rules: ^ 
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(6) 



^In the product rule for erasure decoding, the symbol # denotes a contradiction which is impossible 
if only erasures and no errors occurred. 

^In the product rule for erasure quantization, the symbol # denotes a contradiction. If a contradiction 
is generated then quantizing the given source with no distortion is impossible and the algorithm fails. 
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Specifically, the outgoing message from a + node in a graph like Fig. |21 is computed by 
combining incoming messages from all other edges with the + rule. The outgoing message 
from an = node is computed by combining incoming messages from all other edges with 
the X rule. Whenever an = node has all incoming messages being *, the value of the node 
is arbitrary. This tie can be broken by arbitrarily choosing a value of or 1 provided 
the tie is broken consistently. Essentially, the requirement of consistent tie-breaking can 
be interpreted as a constraint on the message-passing schedule: tie-breaking information 
for a given tie should be propagated through the graph before other ties are broken. 

In order to provide a precise algorithm for the purpose of proving theorems, we 
consider the ERASURE-QUANTIZE in Table |21 based on applying the rules in ® with a 
sequential schedule and all tie-breaking collected into step 8. 

Table 2: An algorithm for iteratively quantizing a source with erasures. 



ERASURE-QUANTIZE(G,z) 

1: while has at least one unerased sample do 

2: if 3 row i of G (i.e., a variable) connected to exactly one unerased check Zj then 
3: Reserve message variable i to later satisfy Zj and erase check zj 
4: else 

5: return FAIL 
6: end if 
7: end while 

8: Arbitrarily set all unreserved message variables 

9: Set reserved variables to satisfy the corresponding checks starting from the last reserved 

variable and working backward to the first reserved variable 
10: return message variables w 



3.4.3 Iterative Algorithm Duality 

Our main results regarding iterative quantization are the following three theorems stating 
that ERASURE-QUANTIZE works correctly, can be analyzed in the same manner as the 
ERASURE-DECODE algorithm, and works quickly: 

Theorem 3. For any linear code with generator matrix G, ERASURE-QUANTIZE(G, i*) 

either fails in step 5 or else returns w such that wG matches z in all unerased positions. 

Theorem 4. Consider a linear code with parity check matrix H and its dual code with 
generator matrix G"*" = H. The algorithm ERASURE-DECODE(H, y) fails in step 5 if and 
only if the algorithm ERASURE-QUANTIZE(G-'", i*) fails in step 5 where y has erasures 
specified by e and z has erasures specified by e-^ = 1 — e. 

Theorem 5. The algorithm ERASURE-QUANTIZE(G, i*) runs in time 0{n ■ d) where n 
is the length of z and d is the maximum degree of the graph corresponding to G. 

These results (proved in Appendix^ imply that the parallel structure between era- 
sure decoding and erasure quantization allows us to directly apply virtually every result 
from the analysis of one to the other. For example, these theorems combined with the 
analysis/design of irregular LDPC codes achieving the capacity of the binary erasure 
channel pi immediately yield the following Corollary: 

Corollary 2. There exists a sequence of linear codes which can he efficiently encoded and 
decoded that achieves the rate- distortion function for binary erasure quantization. 
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4 Concluding Remarks 



In this paper we demonstrated how codes on sparse graphs combined with iterative decod- 
ing can achieve the Shannon hmit for binary erasure quantization. The main contribution 
of our algorithm is in recognizing the role of tie-breaking, scheduling, and initialization 
in iterative quantization. The key insight in our analysis is the strong dual relationship 
between error correction and quantization for codes on graphs and their associated de- 
coding/quantization algorithms (both optimal and iterative). We conjecture that the 
main task in designing iterative message-passing algorithms for more general quantiza- 
tion problems lies in designing appropriate tie-breaking, scheduling, and initialization 
rules for such scenarios and exploiting similar dual relationships to channel decoding. 

A Proofs 

Proof of TheoremUi' Consider quantizing a random source and choose some c > and 
let d be the smallest integer such that at least c • n parity checks have degree at most d. 
For each such parity check, the probability that all variables in the check are not erased 
is at least (1 — e)*^. Hence the probability that the check cannot be satisfied is at least 
(1/2) ■ (1 — eY- Since there are c ■ n such checks, the probability that at least one check 
cannot be satisfied is 

Pr[encoding failure] > 1 - [l - (1/2) ■ (1 - e)'^j ^" (7) 

> 1- [l- (l/2-e/2)^j'" (8) 
= l-exp|c-nln 1 - (1/2 - 6/2)^^ | (9) 

> l-exp|-c-n-(l/2-e/2)'^| (10) 
= l-exp{-exp[lnc + lnn + rfln(l/2-e/2)]}. (11) 

Hence for the probability of decoding failure to become small, d must grow at least 
logarithmically with n for every c > 0. Note that this argument applies to any parity- 
check graph representation of the code. □ 

Proof of Theorem\^ We will show that unique channel decoding is possible if and only if 
the matrix equation Mx = y has a solution (where M will be defined shortly). Similarly, 
we will show that source quantization is possible for every z if and only if the matrix 
equation wM = z has a solution for every z. By demonstrating that both conditions are 
satisfied if and only if the same matrix M has rank ra, we will prove the desired result. 

Assume that all erasures occur in the last |e| positions (z.e., e = 0""'^ l'^). ^ This 
incurs no loss of generality since the coordinates of C can always be permuted accordingly 
and the theorem applied to the permuted code and its permuted dual code. Let x 
represent the transmitted signal and let y denote the received signal. Optimal decoding 
corresponds to finding a vector which is a codeword of C and consistent with the unerased 
received values. The requirement that x is a codeword corresponds to the equation 
Hx = where H is the parity check matrix of C. The requirement that x is consistent 

^We use \d\ to denote the number of non-zero values in a (i.e., the weig ht of a) and b" = {b b . . .b). 
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with the received unerased data corresponds to the equation (In-|ei 0)f = y "~ where 
Ij represents a t-hy-t identity matrix and y I represents the sub- vector . . . ,yj). 

Thus successful decoding is possible if and only if the equation 




has a unique solution. According to well-known properties of linear algebra, uniqueness 
is equivalent to the matrix in ()12p having full column rank {i.e., rank n). Note that 
existence of a solution is guaranteed since a codeword was sent and no errors occurred. 

Let represent the source to be quantized with erasure pattern e^. Since we assumed 
that all erasures in e occurred in the last |e| samples, the dual erasure pattern e-^ has 
all erasures occurring in the first le-'"! positions (i.e., = O'^). Optimal decoding 

corresponds to finding a vector w which is a codeword of C and consistent with the 
unerased received values. The former requirement corresponds to the equation vG-^ = w 
where G"*" is the generator matrix of C"*" and ■y is a binary vector of appropriate dimension. 
The latter requirement corresponds to the equation m (l|^x| O) + w = z where m is a 
binary vector chosen to ensure that the first \e'^\ positions {i.e., the erased positions) 
match regardless of w. Thus successful decoding is possible for every z if and only if a 
solution exists for 



for every z. According to well-known properties of linear algebra, existence of a solution 
for every z is equivalent to the matrix in ()13p having full column rank {i.e., rank n). 
Note that uniqueness of a solution is neither guaranteed not required since quantization 
is successful if at least one solution is found. 

Noting that G"*" = H and le-'"! = n — \e\ completes the proof since these conditions 
imply that the matrices in (fT^ and ()13|) are identical. □ 

Proof of Theorem\^ For the algorithm to exit the while loop and reach step 8, every 
unerased element of Zi must have been erased in step 3 and assigned a reserved message 
variable. After a variable is reserved all its checks must be erased. Since checks can 
never changed from erased to unerased, a reserved variable can never again be selected 
in step 2 and thus a variable can never be reserved more than once. 

Thus after the while loop, each unerased position in z has a corresponding reserved 
variable. Hence there exists an assignment of the message variables which result in a 
codeword matching z in the unerased positions. This assignment could be computed via 
brute-force by solving he corresponding system of linear equations, but in Theorem El we 
show that this step can be computed more efficiently. □ 

Proof of Theorem^ The proof relies on the following invariant for steps 1-7 of both 



This condition is trivially true before the algorithm begins and forms the base case for a 
proof by induction. We assume that (fT^ holds at iteration i of steps 1-7 and show that 
it must also hold at iteration i -|- 1. 

First, (fT^ implies that the outcome of step 1 is the same for each algorithm. Next, 
since G^ = H the tests in step 1 and step 2 of ERASURE-DECODE(H, y) and ERASURE- 
QUANTIZE(G"'", i*) yield the same result. Finally, at step 3, yj is unerased while Zj is 




(13) 



algorithms: 



Vj, yj is erased if and only if Zj is unerased. 



(14) 



9 



erased. Therefore, by induction, condition ()14|) is true at every iteration and ERASURE- 
DECODE(H,^) fails at step 5 if and only if ERASURE-QUANTIZE(G^, 2) fails at step 5. 

□ 

Proof of Theorem\^ The while loop executes at most n times. Therefore step 1 requires 
at most 0{n) operations. Consider storing the number of variables with exactly one 
unerased check in a data structure which supports insertion and removal in constant 
time {e.g., a hash table). We can initialize the data structure with 0{d ■ n) operations. 
Removing an element in steps 2 and 3 and updating the data structure to account for 
step 3 requires 0{d) operations. Thus steps 1 through 8 require 0{d • n) operations and 
all that remains is to bound the running time of step 9. 

Denote the first reserved variable by 'yj(i), the second reserved variable by Vj(2) and so 
on to fj(|e|)- As described in step 9, we first assign a value to Wjdei) and work backward. 
Specifically, we set to the modulo-2 sum of Zj(^i) and all message variables connected 
to Zj{^i) (except fj(j)). This is possible for -Zjdei) since no other reserved variable could be 
connected to -Zj-dei)-"^ Similarly, Zj(\e\-i) must be connected to only unreserved variables 
as well as perhaps to fjdei) and therefore a value can be determined for Vj{\e\-i). Thus, 
by induction we can determine every Vj. 

Adding up the operations computed for each step yields a running time of 0{d ■ n). 

□ 
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''if ^j(|e|) was connected to another reserved variable v, that would imply v was reserved when con- 
ted to 2j(|e|) which was unera 
step 2 in ERASURE-QUANTIZE. 



nected to 2j(|e|) which was unerased as well as z which must also have been unerased. This contradicts 
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